Intrinsic Geometry of Stochastic Gradient Descent Algorithms

We consider the intrinsic geometry of stochastic gradient descent (SG) algorithms. We show how to derive SG algorithms that fully respect an underlying geometry which can be induced by either prior knowledge in the form of a preferential structure or a generative model via the Fisher information metric. We show that using the geometrically motivated update and the “correct” loss function, the implicit and explicit discrete time updates are, under certain conditions, identical. This new loss function reduces to least square loss for linear regression with Gaussian measurement noise. We also show that the seemingly obvious requirement that the loss function is convex is not appropriate in non-flat geometries. We illustrate the power of the new framework by deriving an algorithm for a regression problem over a multinomial distribution.